A Semi-Automatic Evaluation Scheme: Automated Nuggetization for Manual Annotation
نویسندگان
چکیده
In this paper we describe automatic information nuggetization and its application to text comparison. More specifically, we take a close look at how machine-generated nuggets can be used to create evaluation material. A semiautomatic annotation scheme is designed to produce gold-standard data with exceptionally high inter-human agreement.
منابع مشابه
A Semi-automatic Evaluation Scheme
The evaluations of many NLP applications share a two-step procedure: the processing and aggregation of reference data to yield a gold-standard data set, and the subsequent comparison with peer system results. Traditionally, the former has been performed by human annotators and the latter has been conducted either manually or automatically. Moving toward a fully automated evaluation procedure, w...
متن کاملIdentifying Nuggets of Information in GALE Distillation Evaluation
This paper describes an approach to automatic nuggetization and implemented system employed in GALE Distillation evaluation to measure the information content of text returned in response to an open-ended question. The system identifies nuggets, or atomic units of information, categorizes them according to their semantic type, and selects different types of nuggets depending on the type of the ...
متن کاملAutomatic measurement of instantaneous changes in the walls of carotid artery with sequential ultrasound images
Introduction: This study presents a computerized analyzing method for detection of instantaneous changes of far and near walls of the common carotid artery in sequential ultrasound images by applying the maximum gradient algorithm. Maximum gradient was modified and some characteristics were added from the dynamic programming algorithm for our applications. Methods: The algorithm was evaluat...
متن کاملDiscourse Annotation In The Monroe Corpus
We describe a method for annotating spoken dialog corpora using both automatic and manual annotation. Our semi-automated method for corpus development results in a corpus combining rich semantics, discourse information and reference annotation, and allows us to explore issues relating these.
متن کاملAn interactive tool for manual, semi-automatic and automatic video annotation
The annotation of image and video data of large datasets is a fundamental task in multimedia information retrieval and computer vision applications. The aim of annotation tools is to relieve the user from the burden of the manual annotation as much as possible. To achieve this ideal goal, many different functionalities are required in order to make the annotations process as automatic as possib...
متن کامل